Category

This change is exactly one of the following (please change [ ] to [x]) to indicate which:

• a bug fix (Fix #...)
• a new Ripper
• a refactoring
• a style change/fix
• a new feature

Description

This feature adds support to use Redis as the mechanism for skipping already downloaded URLs. If you use RipMe over a longer period of time, to download many, many galleries and albums, the url_history.txt file gets quite large. Doing an O(n) scan through the entire list for every URL in a job becomes VERY expensive. My own url_history.txt file is approaching 3 million lines and 130 MB. Using Redis speeds up the ripping process considerably AND allows power users the ability to coordinate jobs running across multiple machines on a network.

Users can optionally add the following lines to the rip.properties file:

url_history.redis_cache.host = 192.168.0.123  #IP address or domain name for the redis host 
url_history.redis_cache.port = 6379 #redis port (optional, rip me defaults to 6379)
url_history.redis_cache.key_prefix = RipMeURL:  #a prefix to give the keys added to redis (optional, will default to an empty string)

If users do not add this configuration, the URL matching algorithm now uses a HashSet. This is memory intensive, but performs faster than the sequential scan.

Note: RipMe will continue to append new lines to the url_history.txt file since this operation does not seem to slow down the job (...at least at the scales that I have encountered)

Note 2: The easiest way to run redis locally is to use docker (Something like docker run --name my-redis -d -p 6379:6379 redis). Alternatively you could download and install redis for your OS.

Testing

Required verification:

• I've verified that there are no regressions in mvn test (there are no new failures or errors).
• I've verified that this change works as intended.
  • Downloads all relevant content.
  • Downloads content from multiple pages (as necessary or appropriate).
  • Saves content at reasonable file names (e.g. page titles or content IDs) to help easily browse downloaded content.
• I've verified that this change did not break existing functionality (especially in the Ripper I modified).

Optional but recommended:

• I've added a unit test to cover my change.